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Abstract 


This  document  is  a  high-level  overview  of  the  CMM®-Based  Appraisal  for  Internal  Process 
Improvement  (CBA IPI)  VI. 2  assessment  method  and  is  an  update  to  the  CBA IPI  Vl.l 
Method  Description  [Dunaway  96b].  It  provides  a  brief  history  of  SEI  appraisal  methods,  as 
well  as  establishing  appraisals  in  the  context  of  the  IDEAL®^  approach  to  software  process 
improvement.  CBA  IPI  is  a  diagnostic  tool  that  supports,  enables,  and  encourages  an  organiza¬ 
tion’s  commitment  to  process  improvement.  The  method  helps  an  organization  to  gain  insight 
into  its  software  development  capability  by  identifying  strengths  and  weaknesses  of  its  current 
processes  related  to  the  Capability  Maturity  Model®  for  Software  Vl.l.  The  method  focuses 
on  identifying  software  improvements  that  are  most  beneficial,  given  an  organization’s  busi¬ 
ness  goals  and  current  maturity  level.  Brief  descriptions  of  the  method  activities,  roles,  and 
responsibilities  are  provided.  In  addition,  guidelines  are  provided  for  establishing  resource 
requirements  for  conducting  a  CBA  IPI.  The  SEI  Appraiser  Program  is  discussed,  detailing  the 
requirements  for  persons  qualified  to  lead  CBA  IPIs. 


®  CMM  is  registered  with  the  U.S.  Patent  and  Trademark  Office. 
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1  Background  and  Overview 


This  document  provides  a  high-level  overview  of  the  CMM®-Based  Appraisal  for  Internal 
Process  Improvement  (CBA IPI)  VI. 2  method.  The  overview  is  primarily  intended  for  a  spon¬ 
sor  or  an  opinion  leader  in  an  organization  who  is  planning  or  considering  an  assessment. 
Other  audiences  include  potential  team  members,  assessment  participants,  and  individuals 
who  are  involved  in  or  may  be  interested  in  process  improvement.  The  document  addresses 
important  considerations  when  planning  a  CBA  IPI.  This  document  is  not  intended  to  provide 
in-depth  information  to  conduct  a  CBA  IPI  assessment,  perform  detailed  planning  for  an 
assessment,  or  train  an  assessment  team.  Those  wishing  to  conduct  a  CBA  IPI  assessment  in 
their  organization  should  contact  an  SEI-authorized  Lead  Assessor  to  ensure  that  the  assess¬ 
ment  is  performed  by  a  trained,  qualified  individual  who  will  conduct  the  assessment  for  the 
maximum  benefit  of  the  assessed  organization.  A  list  of  SEI-authorized  Lead  Assessors  is 
maintained  on  the  SETs  Web  site  at:  http://www.sei.cmu.edu/managing/app.directory.html. 

In  accordance  with  the  SETs  mission  to  provide  leadership  in  advancing  the  state  of  the  prac¬ 
tice  of  software  engineering  by  improving  the  quality  of  systems  that  depend  on  software, 
there  has  been  strong  emphasis  within  the  Software  Engineering  Institute  (SEI)  on  treating 
software  development  tasks  as  processes  that  can  be  defined,  practiced,  measured,  and 
improved.  In  early  software  process  publications,  a  software  maturity  framework  [Humphrey 
87a]  and  questionnaire  [Humphrey  87b]  were  developed  to  help  organizations  characterize  the 
current  state  of  their  software  practices,  set  goals  for  process  improvement,  and  set  priorities. 

The  SEI  assisted,  a  number  of  organizations  in  performing  assessments  [Olson  89]  based 
largely  on  the  maturity  questionnaire.  This  early  questionnaire  provided  a  scoring  mechanism 
for  determining  a  project’s  maturity  level.  In  1988-91,  the  SEI  provided  training  to  organiza¬ 
tions  who  wished  to  perform  self-assessments  of  their  software  processes. 


®  CMM  is  registered  with  the  U.S.  Patent  and  Trademark  Office. 
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Based  on  the  success  and  widespread  use  of  the  maturity  framework  and  software  process 
assessments,  Version  1.0  of  the  CMM  for  Software  was  published  in  1991  [Paulk  91a,  Paulk 
91b].  In  1993  the  CMM  was  revised,  and  Version  1.1  was  published.  Various  organizations 
modified  SEI  appraisals  to  reflect  the  CMM;  however,  the  CBA  IPI  method  is  the  first  CMM- 
based  assessment  method  released  by  the  SEI.  CBA  IPI  uses  an  updated  maturity  question¬ 
naire  consistent  with  CMM  V  1.1.  Unlike  the  maturity  questionnaire  released  in  1987,  the  cur¬ 
rent  maturity  questionnaire  does  not  include  a  mechanism  for  scoring  maturity  levels 
[Zubrow  94]. 

Using  the  Capability  Maturity  Model®  for  Software  V.1.1  (CMM)  as  a  reference  model 
[Paulk  93a,  Paulk  93b],  the  SEI  developed  the  CBA  IPI  VI.O  in  1995  for  assessing  an  organi¬ 
zation’s  software  process  capability.  CBA  IPI  V  1.1  was  released  in  1996  and  CBA  IPI  VI. 2 
was  released  in  2001  containing  modifications  for  clarification  and  simplification.  In-depth 
documentation  and  guidance  on  the  CBA  IPI  method  is  contained  in  the  CBA  Lead  Assessor’s 

Guide  VI. 2*  that  is  available  to  qualified  persons  who  are  accepted  into  CBA  Lead  Assessor 
Training. 


1 .1  IDEAL^”^  Approach  to  Process  Improvement 

The  CBA  IPI  method  is  a  diagnostic  tool  used  to  appraise  and  characterize  an  organization’s 
software  processes  and  is  used  in  the  larger  context  of  software  process  improvement.The 

SEFs  recommended  framework  for  software  process  improvement  is  the  IDEAL^^  model, 
shown  in  Figure  1  [Peterson  95,  Radice  94].  The  IDEAL  approach  consists  of  five  phases:  ini¬ 
tiating,  diagnosing,  acting,  establishing,  and  learning.  Appraisals  are  an  integral  part  of  the 
diagnosing  phase  of  the  IDEAL  approach. 

Software  process  is  defined  to  mean  the  system  of  all  tasks  and  the  supporting  tools,  standards, 
methods,  and  practices  involved  in  the  production  and  evolution  of  a  software  product 
throughout  the  software  life  cycle.  It  has  become  widely  accepted  that  the  quality  of  a  (soft¬ 
ware)  system  is  largely  governed  by  the  quality  of  the  process  used  to  develop  and  maintain  it 
[Crosby  79,  Doming  86,  Humphrey  90,  Juran  88]. 


1 .  Dunaway,  Donna  K.  CMMSased  Appraisal  for  Interna!  Process  Improvement  (CBA  IPI)  Lead 
Assessor's  Guide  1^7.2  (CMU/SEI-01-HB-002).  Pittsburgh,  Pa.:  Software  Engineering  Institute, 
Carnegie  Mellon  University,  2001 .  This  document  is  available  only  through  Lead  Assessor 
Training — It  Is  not  available  through  normal  SEI  distribution  channels. 

©  Capability  Maturity  Model  is  registered  with  the  U.S.  Patent  and  Trademark  Office. 

SM  is  a  service  mark  of  Carnegie  Mellon  University. 
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Figure  1:  IDEAL^^  Model 

The  initiating  phase  of  process  improvement  should  be  successfully  undertaken  before  the 
appraisal  start-up.  First,  some  outside  stimulus  for  improvement  needs  to  occur.  Then  sponsor¬ 
ship  is  established  with  the  site’s  senior  management,  and  efforts  toward  building  an  improve¬ 
ment  infrastructure  are  committed.  An  organization  could  be  appraised  for  the  first  time  or 
could  be  reappraising  their  processes  in  preparation  for  the  next  process  improvement  cycle. 
The  IDEAL  approach  assumes  that  the  appraisal  will  be  followed  by  a  thorough  documenta¬ 
tion  of  the  appraisal  results  and  the  development  of  recommendations.  The  software  process 
baseline  established  by  the  appraisal  and  the  recommendations  and  priorities  that  come  from 
the  appraisal  form  a  basis  for  follow-on  activities.  These  activities  are  typically  documented  in 
an  action  plan  for  furthering  process  improvement  activities. 


1 .2  SEI  Appraisal  Methods 

To  provide  a  framework  for  rating  an  organization’s  process  capability  against  the  CMM  and 
to  provide  a  basis  for  comparing  assessment  and  evaluation  results,  the  SEI  published  the 
CMM  Appraisal  Framework  (CAF)  [Masters  95].  The  CAF  identifies  the  requirements  and 
desired  characteristics  of  a  CMM-based  appraisal  method  to  improve  consistency  and  reliabil¬ 
ity  of  methods  and  their  results.  An  appraisal  method  is  CAF  compliant  when  it  meets  all  of 
the  CAF  requirements.  The  term  appraisal  as  used  at  the  SEI  includes  multiple  methods,  such 
as  assessments  and  evaluations,  all  of  which  focus  on  an  organization’s  software  development 
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process.  Both  CBA  IPI  and  Software  Capability  Evaluation  (SCE)  V3.0  [Byrnes  96]  were 
designed  to  be  CAF  compliant. 

1.2.1  CMM-Based  Appraisal  for  Internal  Process  Improvement 
(CBA  IPI) 

The  CBA  IPI  method  was  created  in  response  to  user  needs  for  a  CMM-based  assessment 
method.  The  Software  Process  Assessment  (SPA)  method,  which  became  so  familiar  in  the 
software  community,  pre-dated  the  CMM.  Although  many  organizations  modified  the  SPA  to 
reflect  the  CMM,  there  was  a  wide  range  of  approaches  and  results. 

The  CBA  IPI  method  was  developed  and  field  tested  in  1994.  After  factoring  in  lessons 
learned  from  the  community  feedback,  the  SEI  released  CBA  IPI  V  1.0  in  May  1995.  The 

method  and  documentation  were  upgraded  to  CBA  IPI  V  1.1  in  March  1996“.  The  CBA  IPI 
method  explicitly  uses  CMM  VI.  1  as  a  reference  model.  The  data  collection  is  based  on  key 
process  areas  (KPAs)  of  the  CMM  as  well  as  non-CMM  issues.  CBA  IPI  is  intended  to  estab¬ 
lish  consistency  among  CMM-based  assessments  so  that  results  from  one  assessment  can  be 
compared  to  those  of  another.  The  CBA  IPI  method  complies  with  the  CAF,  so  results  from  a 
CBA  IPI  are  intended  to  be  consistent  with  results  from  other  CAF-compliant  methods.  The 
CBA  IPI  method  is  described  in  more  detail  in  the  following  sections  of  this  report.  Appendix 
B  contains  descriptions  of  the  method  roles  and  responsibilities,  while  Appendix  C  contains  a 
glossary  of  terms  commonly  used  in  a  CBA  IPI. 

1.2.2  Software  Capability  Evaluation  (SCE) 

SCEs  are  used  for  software  acquisition  as  a  discriminator  to  select  suppliers,  for  contract  mon¬ 
itoring,  and  for  incentives.  They  can  also  be  used  for  evaluation  of  internal  processes.  SCE 
V2.0  [CBA  Project  94]  was  updated  to  reflect  CMM  Vl.l.  SCE  V3.0  is  CAF  compliant. 
Therefore,  results  from  an  SCE  V3.0  should  be  consistent  with  a  CBA  IPI  Vl.l  if  the  areas  of 
investigation  are  the  same  and  in  relatively  the  same  time  frame.  SCE  is  used  to  gain  insight 
into  the  software  process  capability  of  a  supplier  organization  and  is  intended  to  help  decision 
makers  make  better  acquisition  decisions,  improve  subcontractor  performance,  and  provide 
insight  to  a  purchasing  organization. 


2.  Dunaway,  Donna  K.  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CBA  IPI)  Lead 
Assessor's  Guide  V1.1  (CMU/SEI-96-HB-003).  Pittsburgh,  Pa.:  Software  Engineering  Institute, 
Carnegie  Mellon  University,  1996.  This  document  is  available  only  through  Lead  Assessor 
Training — it  is  not  available  through  normal  SEI  distribution  channels. 
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1 .2.3  Differences  Between  Assessments  and  Evaluations 


The  basic  difference  between  an  assessment  and  an  evaluation  is  that  an  assessment  is  an 
appraisal  that  an  organization  does  to  and  for  itself;  an  evaluation  is  an  appraisal  where  an 
external  group  comes  into  an  organization  and  looks  at  the  organization’s  process  capability  to 
make  a  decision  regarding  future  business.The  scope  of  an  assessment  is  determined  relative 
to  the  organization’s  needs  and  the  business  goals  of  the  sponsor,  who  is  usually  the  senior 
manager  of  the  assessed  organization.  In  contrast,  the  scope  of  an  evaluation  is  determined  rel¬ 
ative  to  the  needs  of  the  sponsor,  who  is  the  individual  or  individuals  responsible  for  deciding 
to  conduct  the  evaluation  of  the  organization(s)  with  whom  the  sponsor  is  currently  or  consid¬ 
ering  doing  business. 

After  an  assessment,  the  senior  manager  of  the  assessed  organization  owns  the  assessment 
findings  and  results,  and  generally  uses  the  results  to  formulate  an  action  plan  for  the  process 
improvement  program.  After  an  evaluation,  the  sponsor  owns  the  evaluation  results  and  uses 
the  results  to  make  decisions  regarding  the  recipient  organization(s),  such  as  source  selection, 
incentive  fee  awards,  performance  monitoring,  risk  management,  and  measurement  of  internal 
improvement;  these  results  may  or  may  not  be  shared  with  the  evaluated  organization. 

Assessments  are  intended  to  motivate  organizations  to  initiate  or  continue  software  process 
improvement  programs.  Evaluations  are  externally  imposed  motivation  for  organizations  to 
undertake  process  improvement.  Assessment  teams  represent  a  collaboration  among  team 
members,  many  of  whom  are  from  the  organization  being  assessed.  Evaluation  teams,  on  the 
other  hand,  take  more  of  an  audit-oriented  approach  with  a  more  formal  interface  between 
team  members  and  organization  representatives;  evaluation  teams  rarely  include  representa¬ 
tives  from  the  evaluated  organization. 


1 .2.4  interim  Profile 

Interim  Profile  [Whitney  94]  is  a  method  to  rapidly  measure  the  status  of  an  organization’s 
software  engineering  process  improvements  between  organizational  assessments  and  was  not 
intended  to  comply  with  the  CAP. 
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1.2.5  Standard  Assessment  Method  for  Process 

Improvement  (SCAMPI®'^) 

The  SCAMPI^'^  method  Vl.O  is  based  on  the  same  assessment  principles  as  mentioned  for  a 
CBA  IPI.  The  SCAMPI  method  [SEI  00a]  is  designed  to  be  a  Class  A  method  and  satisfies 
each  of  the  requirements  stated  in  the  Assessment  Requirements  for  CMMI  (ARC)  [SEI  00b] . 
The  method  uses  the  Capability  Maturity  Model  -  Integrated  for  Systems  Engineering  and 
Software  Engineering  (CMMI-SE/SW)  model  [SEI  00c,  SEIOOd]  as  a  reference  model.  The 
method  was  developed  over  the  1999-2000  time  frame  along  with  development  of  the  inte¬ 
grated  CMM.  Its  use  is  expected  to  grow  as  organizations  become  familiar  with  the  integrated 
model. 


1 .3  Results  of  Process  Improvement  Efforts 

The  initial  state  of  the  practice  reported  on  assessment  data  largely  from  project-based  ques¬ 
tionnaire  data  from  10  sites  [Humphrey  89].  The  follow-up  report  included  the  first  site-based 
software  process  maturity  profile  and  provided  an  analysis  of  assessment  findings  from  59 
sites  showing  the  frequency  with  which  key  process  deficiencies  were  identified  by  assess¬ 
ment  teams  [Kitson  92].  Another  report  presents  an  analysis  of  assessment  results  from  48 
organizations  that  have  performed  2  or  more  assessments  and  focuses  on  the  time  required  to 
increase  process  maturity  [Hayes  95].  Updates  on  the  process  maturity  profile  of  the  software 
community  are  published  on  the  SEI  Web  site:  http://www.sei.cmu.edu/sema/profile.htmI.  The 
August  2001  Maturity  Profile  presents  data  from  1,483  CBA  IPI  assessments. 

The  SEI  has  published  reports  that  show  a  relationship  between  CMM-based  improvement 
and  organizational  performance.  One  report  [Herbsleb  94]  documents  process  improvement 
efforts  in  13  organizations  by  showing  improvements  in  cycle  time,  defect  density,  and  pro¬ 
ductivity.  Benefit-to-cost  ratios  presented  range  from  4.0:1  to  8.8:1.  In  a  more  comprehensive 
report  on  the  impact  of  CMM-based  appraisals  on  subsequent  software  process  improvement 
and  organizational  performance  [Goldenson  95],  survey  results  from  138  appraisal  partici¬ 
pants  representing  56  appraisals  are  presented.  The  results  show  that,  in  general,  increased 
process  maturity  results  in  better  product  quality,  ability  to  meet  schedule  commitments,  and 
other  indicators  of  organizational  performance. 

Panels  have  been  presented  by  CBA  IPI  users  at  domestic  and  international  conferences  since 
1996.  Five  of  these  panelists  contributed  to  a  technical  report  published  in  1999  [Dunaway  99] 
available  on  the  SEI  Web  site.  The  report  focuses  on  the  organizational  perspective  before, 
during,  and  after  the  assessment. 


CMMI  and  SCAMPI  are  service  marks  of  Carnegie  Mellon  University. 


6 


CMU/SEI-2001-TR-033 


To  determine  user  satisfaction  in  the  assessment  method  as  part  of  their  process  improvement 
initiative,  the  SEI  collects  feedback  at  the  conclusion  of  each  assessment  from  assessment 
sponsors,  assessment  team  members,  and  the  team  leaders.  Results  from  the  feedback  were 
reported  at  the  1996  Software  Engineering  Process  Group  Conference  [Dunaway  96a]  and  the 
1998  Software  Engineering  Process  Group  Conference  [Dunaway  98].  A  November  2001 
report  presents  an  analysis  of  information  received  from  assessment  participants  [Dunaway 
Ola].  Highlights  of  the  report  include  that  78%  of  assessment  sponsors  indicated  the  assess¬ 
ment  findings  were  “very  accurate”  in  reflecting  the  state  of  the  software  process  in  their  orga¬ 
nization  and  more  than  85%  rated  the  assessment  team’s  performance  to  be  “excellent.” 


1 .4  Assessments  and  Benchmarking 

CBA IPI  assessments  are  not  intended  to  be  used  for  certification  purposes.  With  the  increas¬ 
ing  pressure  to  achieve  maturity  levels  being  placed  on  contractors  and  subcontractors,  it  is 
important  to  be  specific  as  to  how  a  maturity  level  is  measured  and  used  as  a  benchmark 
within  the  industry.  These  questions  are  to  help  those  who  need  to  report  their  assessment 
results  as  well  as  those  who  need  to  rely  on  what  an  organization  says  about  its  assessment 
results  and  maturity  level. 

What  was  the  date  and  location(s)  of  the  assessment? 

What  was  the  organizational  scope  of  the  assessment?  For  any  large  organization,  only  a 
part  is  usually  selected  for  the  scope  of  the  assessment.  Assessment  results  do  not  extrapolate 
beyond  the  part  of  the  organization  being  investigated. 

What  was  the  CMM  scope  of  the  assessment?  Which  reference  model  was  used  in  the 
assessment  and  which  process  areas  within  the  model  comprised  the  assessment  scope? 

What  assessment  method  was  used?  There  are  many  assessment  methods;  some  are  more 
comprehensive  than  others.  When  a  claim  relative  to  a  maturity  level  is  being  asserted,  a  high 
degree  of  confidence  in  the  results  is  needed. 

Who  conducted  the  assessment?  Was  the  assessment  team  leader  a  trained,  experienced  per¬ 
son  qualified  to  conduct  the  method  used?  Did  the  team  members  meet  the  selection  criteria 
and  receive  the  required  training? 

May  I  see  the  assessment  results?  For  anyone  who  is  asked  to  accept  an  organization’s  matu¬ 
rity  level  for  the  purpose  of  doing  future  business,  they  have  a  right  to  ask  to  see  the  assess¬ 
ment  findings  including  the  strengths  and  weaknesses  for  each  process  area  examined. 
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1.5  Role  of  the  SEI  in  CMM-Based  Assessments 


The  SEI  trains  and  authorizes  qualified  persons  to  lead  appraisals,  e.g.,  CBA  IPI,  and  to  use 
SEI  materials  in  conducting  the  appraisals.  However,  the  SEI  exercises  quality  control  over  the 
use  of  assessment  material. 

The  SEI  requires  authorized  Lead  Assessors  for  CBA  IPI  to  return  a  report  to  the  SEI  for  each 
completed  assessment.  The  data  is  entered  into  the  Process  Appraisal  Information  System 
(PAIS).  All  assessment  data  returned  to  the  SEI  are  kept  confidential. 

Assessment  data  are  reported  in  industry  aggregates  with  no  identification  of  organizations 
and  projects.  The  Maturity  Profile  Report  is  published  by  the  SEI  twice  a  year  and  is  available 
on  the  SEI  Web  site  (http://www.sei.cmu.edu)  as  well  as  through  SEI  Customer  Relations  (see 
page  44  for  contact  information). 

The  SEI  does  not  “certify”  any  assessment  findings  or  maturity  levels.  The  SEI  does  not  con¬ 
firm  or  deny  that  any  organization  has  performed  an  assessment.  The  assessment  sponsor 
owns  the  assessment  findings  and  maturity  level.  The  sponsor  may  publicize  this  information 
at  their  discretion.  The  SEI  does  not  validate  or  certify  appraisal  results. 

The  SEI  collects  feedback  data  from  assessment  participants  to  understand  how  well  the 
assessment  was  performed  and  to  identify  areas  for  improvement.  Summaries  of  the  feedback 
data  from  assessment  sponsors,  assessment  team  members,  and  assessment  team  leaders  is 
published  periodically.  The  data  are  presented  without  attribution  to  organizations  or  projects 
as  in  the  Maturity  Profile. 
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2  What  Is  the  CBA  IPI  Method? 


The  CBA  IPI  method  is  a  diagnostic  tool  that  enables  an  organization  to  gain  insight  into  its 
software  development  capability  by  identifying  strengths  and  weaknesses  of  its  current  pro¬ 
cesses,  to  relate  these  strengths  and  weaknesses  to  the  CMM,  to  prioritize  software  improve¬ 
ment  plans,  and  to  focus  on  software  improvements  that  are  most  beneficial,  given  its  current 
level  of  maturity  and  the  business  goals. 

The  method  is  an  assessment  of  an  organization’s  software  process  capability  by  a  trained 
group  of  professionals  who  work  as  a  team  to  generate  findings  and  ratings  relative  to  the 
CMM  key  process  areas  within  the  assessment  scope.  The  findings  are  generated  from  data 
collected  from  questionnaires,  document  review,  presentations,  and  in-depth  interviews  with 
middle  managers,  project  leaders,  and  software  practitioners. 

The  CBA  IPI  method  has  two  primary  goals: 

•  to  support,  enable,  and  encourage  an  organization’s  commitment  to  software  process 
improvement 

•  to  provide  an  accurate  picture  of  the  strengths  and  weaknesses  of  the  organization’s 
current  software  process,  using  the  CMM  as  a  reference  model,  and  to  identify  key  process 
areas  for  improvement 

The  approach  of  the  CBA  IPI  method  is  to  assemble  and  train  a  competent  assessment  team 
under  the  leadership  of  a  Lead  Assessor  and  to  conduct  a  structured  series  of  activities  with 
key  people  in  the  organization  to  understand  their  problems,  concerns,  and  ideas  for  improve¬ 
ment.  The  method  is  based  on  the  following  key  assessment  principles: 

•  Use  the  Capability  Maturity  Model  for  Software  Vl.l  as  a  process  reference  model. 

•  Use  a  formalized  assessment  process  that  complies  with  the  CMM  Appraisal  Framework. 

•  Involve  senior  management  as  the  assessment  sponsor. 

•  Base  the  assessment  on  the  sponsor’s  business  goals  and  needs. 

•  Observe  strict  confidentiality  by  guaranteeing  that  no  information  will  be  attributed  to  an 
individual  or  project. 

•  Approach  the  assessment  as  a  collaboration  between  the  assessment  team  and  the 
organizational  participants. 
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2.1  What  Can  This  Method  Accomplish? 


The  business  needs  for  process  improvement  drive  the  requirements  for  an  assessment.  The 
business  needs  for  process  improvement  generally  include  one  or  more  of  three  closely  related 
factors:  reducing  costs,  improving  quality,  and  decreasing  time  to  market.  The  fundamental 
assumption  is  that  these  factors  are  largely  determined  by  the  development  processes. 

Since  the  CBA IPI  method  is  designed  to  support  a  variety  of  assessment  needs,  the  results  of 
a  CBA  IPI  can  support  a  variety  of  activities  that  require  detailed  knowledge  about  the 
strengths  and  weaknesses  of  the  software  process,  such  as 

•  establishing  software  process  action  plans 

•  measuring  actual  progress  against  action  plans 

•  identifying  best  (most  effective)  practices  within  the  organization  for  transition  elsewhere 
in  the  organization 


The  CBA  IPI  method  must  build  on  an  organization’s  commitment  established  during  previous 
phase(s)  of  the  software  process  improvement  cycle.  To  support  the  goal  of  enabling  and  sup¬ 
porting  an  organization’s  commitment  to  process  improvement,  the  assessment  team  and 
assessment  participants  must  understand  the  sponsor’s  business  goals  and  allow  for  a  collabo¬ 
rative  shaping  of  the  assessment  scope.  It  is  important  that  the  organization  owns  and  “buys 
in”  to  the  assessment  results,  identifying  the  most  critical  issues  (CMM  and  non-CMM)  that 
need  to  be  addressed  to  sustain  momentum  for  the  follow-on  activities. 

The  other  primary  goal  of  the  CBA  IPI  is  to  provide  an  accurate  picture  of  existing  software 
processes.  To  accomplish  this,  the  assessment  team  will 

•  provide  the  organization  with  the  data  needed  to  baseline  the  organization’s  software 
capability 

•  identify  major  non-CMM  issues  that  have  an  impact  on  process  improvement  efforts 

•  identify  strengths  and  weaknesses  using  the  CMM  as  a  reference  model 

•  provide  findings  to  the  sponsor  and  the  organization  that  are  sufficiently  complete  to  guide 
the  organization  in  planning  and  prioritizing  future  process  improvement  activities 


2.2  Minimum  Requirements  for  a  CBA  IPI 

For  an  assessment  to  be  considered  a  CBA  IPI,  current  assessment  material  must  be  obtained 
from  the  SET  The  assessment  must  satisfy  the  following  minimum  requirements  concerning 
the  assessment  team,  assessment  plan,  data  collection,  data  validation,  rating,  and  reporting  of 
assessment  results.  Permissible  tailoring  options  are  provided  with  the  requirements. 
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2.2.1  Assessment  Team 


The  assessment  team  must  satisfy  the  following  requirements: 

•  The  assessment  team  must  be  led  by  an  authorized  SEI  Lead  Assessor. 

•  The  team  shall  consist  of  a  minimum  of  4  and  a  maximum  of  10  team  members.  At  least  1 
team  member  must  be  from  the  organization  being  assessed. 

•  All  team  members  must  receive  the  SETs  Introduction  to  the  CMM  course,  or  its 
equivalent,  and  the  SEI’s  CBA IPI  Team  Training  course. 

•  Team  members  must  meet  the  selection  guidelines  relative  to  software  engineering, 
management,  and  assessment  experience. 

Tailoring  options  are  as  follows: 

•  the  size  of  the  assessment  team,  as  long  as  the  team  consists  of  between  4  and  10  qualified 
individuals. 

•  the  composition  of  the  team  as  to  whether  they  are  internal  or  external  to  the  organization, 
as  long  as  one  team  member  is  from  the  organization  being  assessed. 


2.2.2  Assessment  Plan 

An  assessment  plan  must  be  created  that,  at  a  minimum,  contains  the  following: 

•  the  sponsor’s  business  goals  for  the  assessment 

•  the  CMM  scope  (KPAs  to  be  examined) 

•  the  organizational  scope  (the  part  of  the  organization  being  assessed)  including  selected 
projects  and  assessment  participants 

•  a  schedule  for  assessment  activities  and  identification  of  the  resources  to  perform  the 
activities 

•  the  assessment  outputs  and  any  anticipated  follow-on  activities 

•  planned  tailoring  of  the  assessment  method 

•  risks  and  constraints  associated  with  execution  of  the  assessment 

•  the  sponsor's  approval  of  the  plan  and  authorization  for  the  assessment  to  be  conducted. 

Tailoring  options  are  as  follows: 

•  weighting  of  the  assessment  goals.  Depending  upon  the  organization’s  motivation  for 
having  an  assessment,  more  emphasis  may  be  placed  on  one  goal  than  the  other,  e.g.,  more 
focus  toward  supporting  an  organization's  software  process  improvement  than  precise 
conformance  relative  to  the  CMM. 
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•  the  specific  organizational  entities  that  comprise  the  organization’s  scope  for  the 
assessment 

•  the  specific  KPAs  selected  that  comprise  the  CMM  scope  for  the  assessment 

•  the  number  of  projects  and  their  particular  characteristics 

•  the  length  of  time  for  the  assessment 

2.2.3  Data  Collection 

Assessment  data  is  collected  from  four  data  collection  sources  (instruments,  presentations, 
interviews,  and  documents)  and  at  a  minimum  includes  the  following: 

•  instrument  data  (maturity  questionnaire  responses)  from  at  least  the  project  leaders  from 
the  selected  projects 

•  interview  data  from  project  leaders  from  selected  projects  via  individual  interviews 

•  interview  data  from  functional  area  representatives  (practitioners)  and  middle  managers 
via  group  interviews 

•  data  from  documents  for  each  of  the  KPA  goals  within  the  CMM  scope  of  the  assessment 

•  data  from  presentations  via  a  review  of  draft  findings  with  the  assessment  participants 

Conduct  each  data  collection  activity  with  all  assessment  team  members  participating  as  a 
group. 

Maintain  confidentiality  of  all  data  sources  without  attribution  to  persons  or  projects,  unless 
initial  goals  are  specified  to  maintain  project-specific  data.Tailoring  options  are  as  follows: 

•  Collect  instrument  data  from  more  respondents  than  from  the  project  leaders. 

•  Collect  the  site  information  packet. 

•  Conduct  a  project  leader  interview  with  more  than  one  project  representative  if  the  project 
leadership  is  divided  among  multiple  persons. 

•  Conduct  part  of  a  group  interview  “free  form”  where  interviewees  are  asked  to  discuss 
anything  they  believe  the  assessment  team  should  know. 

•  Use  individuals  or  mini-teams  for  document  review,  consolidation  tasks,  and  follow-up 
interviews 

•  Increase  the  emphasis  on  collecting  document  data. 

•  Vary  the  number  of  draft  findings  presentations  that  are  held. 
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2.2.4  Data  Validation 


Data  must  be  validated  using  the  rules  of  corroboration.  The  rules  of  corroboration  are  as  fol¬ 
lows: 

•  Observations^  are  based  on  data  from  at  least  two  independent  sources,  e.g.,  two  separate 
people  or  a  person  and  a  document. 

•  Observations  are  based  on  data  obtained  during  at  least  two  different  data  gathering 
sessions. 

•  Observations  are  confirmed  by  at  least  one  data  source  reflecting  work  actually  being 
done,  e.g.,  an  implementation  level  document  or  an  interview  with  a  person  who  is 
performing  the  work. 

Data  must  sufficiently  cover^  each  CMM  key  practice  within  the  assessment  scope  as  con¬ 
firmed  by  the  entire  assessment  team. 

Tailoring  options  are  as  follows: 

•  extent  of  documentation  that  is  collected 

2.2.5  Rating 

Ratings  must  be  based  on  the  CAP  criteria  for  rating  the  process  maturity  of  an  organization 

against  the  CMM. 

Use  consensus  as  the  decision-making  mechanism  throughout  the  assessment  process. 

Tailoring  options  are  as  follows: 

•  Add  a  “partially  satisfied”  rating  that  would  translate  to  “unsatisfied”  for  maturity  level 
rating. 

•  Extend  ratings  to  common  features  and/or  key  practices. 

•  Rate  the  organization's  maturity  level. 


3.  Observations  are  statements  that  represent  the  assessment  team  members’  understanding  of 
information  either  seen  or  heard  during  the  data  collection  activities. 

4.  A  CMM  key  practice  is  considered  to  have  sufficient  data  coverage  If  the  data  collected  are 
adequate  to: 

-understand  the  extent  of  Implementation  of  the  practice 
-represent  the  organizational  scope  of  the  assessment 
-represent  the  life  cycle  phases  in  use  within  the  assessment  scope 
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2.2.6  Reporting  the  Results 


A  final  findings  briefing  must  be  given  to  the  sponsor  that  presents  the  strengths  and  weak¬ 
nesses  of  each  KPA  within  the  assessment  scope,  as  well  as  a  KPA  profile  that  indicates 
whether  KPAs  are  satisfied,  unsatisfied,  not  rated,  or  not  applicable. 

A  confidence  report  is  given  to  the  sponsor  either  verbally  or  in  writing  to  point  out  any  areas 
that  would  produce  particularly  high  or  low  confidence  in  accuracy  of  the  assessment 
results.Tailoring  options: 

•  Include  consequences  and/or  recommendations  with  the  assessment  findings. 

•  Generate  a  written  final  report  that  details  the  assessment  findings. 

•  Produce  project-specific  reports  if  initially  stated  as  a  goal  of  the  assessment. 

The  Lead  Assessor  is  required  to  return  to  the  SEI  the  following  data  (in  English)  within  30 
days  after  the  conclusion  of  each  assessment: 

•  PAIS  report 

•  Final  Findings  Briefing,  along  with  KPA  profile  indicating  KPAs  investigated  and  their 
satisfaction  level 

•  Assessment  plan  with  planned/actual  schedule  of  assessment  activities 

•  Feedback  forms  from  assessment  team  members,  team  leader,  and  sponsor 

•  Lead  Assessor’s  Requirements  Checklist 


2.3  Differences  Between  CBA IPI  V1.1  and  V1.2 

Changes  to  the  CBA  IPI  method,  the  SW-CMM  Lead  Assessor  Track  in  the  SEI  Appraiser 

Program,  and  changes  to  Lead  Assessor’s  Guide  Vl.l  were  motivated  by  the  following: 

•  evaluation  of  change  requests  received  from  the  user  community  and  reviewed  by  the 
change  control  board 

•  questions  emanating  from  misunderstanding  or  ambiguity  where  simplification  and 
clarification  were  needed 

•  feedback  from  the  user  community  evidencing  inconsistent  interpretation  of  method 
features  where  more  explicit  requirements  were  needed  for  consistency  of  assessment 
results. 

2.3.1  CBA  IPI  Method  Changes 

Changes  to  the  CBA  IPI  method  are  as  follows: 
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•  Data  must  be  collected  for  each  key  practice  of  each  of  the  common  features  instead  of 
permitting  a  sampling  of  data  for  institutionalization  of  common  features.  Experience  has 
shown  that  data  at  the  level  of  detail  of  the  key  practices  was  needed  to  have  a  good 
understanding  of  whether  the  institutionalization  common  feature  was  adequately 
implemented.  Most  Lead  Assessors  have  reported  that  they  collect  data  for  each  key 
practice  in  spite  of  the  sampling  permitted  in  CBA IPI  Vl.l. 

•  The  provision  for  excluding  an  activity  from  coverage  in  the  “activities  performed” 
common  feature  has  been  deleted  from  the  method.  There  were  rare  occurrences  of  an 
activity  being  denoted  “EC”  when  assessment  reports  were  returned  to  the  SET 
Discussions  with  the  user  community  have  indicated  that  it  is  more  valuable  to  have  an 
observation  related  to  each  activity  indicating  either  an  alternative  practice  or  some  reason 
that  the  activity  is  not  performed,  rather  than  being  silent  on  the  issue  by  using  an  “EC.”  If 
a  key  practice  is  determined  not  significant  in  the  organization’s  environment,  which  was 
the  criteria  for  EC,  then  any  weakness  related  to  the  practice  will  not  be  considered  to  have 
a  negative  impact  on  the  associated  goal. 

2.3.2  SW-CMM  Lead  Assessor  Track 

Changes  to  the  SW-CMM  Lead  Assessor  Track  of  the  SEI  Appraiser  Program  are  as  follows: 

•  As  part  of  a  Lead  Assessor’s  responsibilities  during  a  two-year  authorization  term,  he/she 
must  not  only  participate  (as  a  team  leader  or  team  member)  in  two  CBA  IPIs  but  must 
serve  as  the  assessment  team  leader  for  at  least  one  of  the  CBA  IPIs  on  which  they 
participate. 

•  The  requirements  and  responsibilities  for  an  Observing  Lead  Assessor  have  been 
expanded  and  clarified  (see  Appendix  A  for  details). 

2.3.3  Supplements  to  Guidance 

The  following  additional  guidance  topics  are  included  in  the  Lead  Assessor’s  Guide  VI.  1.2 

•  It  is  strongly  recommended  that  the  schedule  of  assessment  activities  limit  the  planned 
assessment  workday  to  a  ten-hour  day  maximum.  Reports  of  long  days  and  late  nights 
have  resulted  in  negative  situations  with  less  than  valuable  assessment  results. 

•  The  guidance  section  on  rating  has  been  re-written  to  clarify  that  only  a  KPA  may  be 
deemed  “not  applicable”  for  an  organization’s  assessment  scope.  If  a  KPA  is  applicable, 
then  all  of  its  goals  and  practices  are  applicable. 

•  A  section  has  been  added  into  Part  1  that  discusses  benchmarking  where  maturity  levels 
are  used  to  compare  an  organization  with  others  in  the  industry.  Details  are  added  that 
clarify  the  meaning  of  a  maturity  level  relative  to  the  assessment’s  organizational  and 
model  scope.  When  making  a  maturity  level  claim,  an  organization  should  report  the 
assessment  scope  (both  CMM  and  organization),  reference  model  used,  assessment 
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method  used,  assessment  date,  and  the  names  of  the  assessment  team  leader  and 
assessment  team  members. 

•  Discussion  is  included  to  clarify  the  role  of  the  Software  Engineering  Institute  in 
assessments.  This  is  to  clarify  the  fact  that  the  SEI  does  not  “certify”  assessment  results. 

•  Discussion  of  the  Transition  Partner  Web  Center  <http://tpweb.sei.cmu.edu>  has  been 
added,  including  assessment  setup,  access  to  assessment  materials,  and  online  entry  of 
assessment  feedback. 


16 


CMU/SE1-2001-TR-033 


3  Who  Participates  in  a  CBA  IPI? 


To  ensure  that  a  CBA  IPI  has  a  successful  impact  on  the  assessed  organization,  many  factors 
need  careful  consideration.  Foremost,  the  person  who  leads  the  assessment  team  must  be  well 
qualified  and  trained.  In  addition,  members  of  the  assessment  team  must  meet  certain  criteria 
for  the  team  composition.  Furthermore,  the  selection  of  the  assessment  participants  must  ade¬ 
quately  cover  the  scope  of  the  assessed  organization. 


3.1  Who  Is  Qualified  to  Lead  a  CBA  IPI? 

A  CBA  IPI  must  be  led  by  an  authorized  Lead  Assessor  in  the  SEI  Appraiser  Program.  Experi¬ 
ence  in  performing  various  assessments  has  proven  that  it  is  critical  for  assessments  to  be  led 
by  qualified  and  well-trained  individuals  to  achieve  accurate  results  in  a  reasonable  amount  of 
time  and  achieve  a  successful  organization  intervention.  These  individuals  must  have  software 
development  experience,  CMM  knowledge  and  experience,  and  training  in  assessment  tech¬ 
nology. 

By  having  well-trained  and  qualified  assessors,  different  assessors  should  get  similar  results  in 
organizations  of  similar  maturity.  The  SEI  Appraiser  Program  is  designed  to  maintain  the 
quality  of  appraisal  leaders  within  the  software  community.  The  SEI  strives  to  ensure  the  con¬ 
tinued  confidence  of  the  U.S.  software  community  in  the  quality  of  SEI  appraisal  technologies. 
Lead  Assessors  are  authorized  within  the  program  to  perform  CBA  IPIs,  using  SEI  copy¬ 
righted  materials.  Lead  Assessors  are  authorized  to  market  and  perform  assessments  either  for 
third-party  organizations  or  for  their  own  organizations’  internal  use.  The  SEI  publishes  the 
SEI  Appraiser  Directory  semi-annually  listing  the  authorized  Lead  Assessors.  A  copy  of  this 
directory  and  the  detailed  requirements  for  a  person  to  qualify  as  a  Lead  Assessor  are  available 
from  SEI  Customer  Relations  (see  page  44  for  contact  information)  and  through  the  SETs  Web 
site  (http://www.sei.cmu.edu).  Appendix  A  describes  the  SEI  Appraiser  Program. 


3.2  What  Are  the  Requirements  for  Assessment  Team 
Members? 

The  size  and  qualifications  of  the  assessment  team,  along  with  any  potential  biases  of  site  team 
members,  may  affect  the  confidence  the  sponsor  has  in  the  assessment  results.  This  is  particu¬ 
larly  true  if  the  team  member  has  been  responsible  for  implementing  process  improvements. 
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Selection  of  the  assessment  team  takes  into  account  the  knowledge,  skills,  and  abilities  of  the 
assessment  team  as  a  whole  as  well  as  each  individual  team  member.  The  team  as  a  whole 
must  have  the  collective  knowledge,  skills,  and  ability  to  conduct  a  CBA  IPI  assessment  for 
the  particular  organization  being  assessed.  Team  members  are  selected  so  that  their  combined 
experience  and  skills  match  what  is  required  for  the  assessment.  Note  that  an  assessment  team 
size  of  eight  individuals  is  recommended  (not  less  than  4  nor  more  than  10)  for  several  rea¬ 
sons.  Smaller  teams  may  have  difficulty  performing  extensive  document  review  and  achieving 
accuracy  in  interviewing  activities.  Larger  teams  require  more  coordination  and  time  to  reach 
consensus  on  the  judgments  that  need  to  be  made  by  the  team  as  a  whole. 

Factors  to  be  considered  in  selecting  assessment  team  members  include  the  following: 

•  knowledge  of  the  CMM.  All  team  members  must  have  successfully  completed  CMM 
training  prior  to  CBA  IPI  team  training.  Team  members  must  have  knowledge  of  each  of 
the  KPAs  within  the  anticipated  maturity  level  and  lower  maturity  levels.  This  knowledge 
includes  the  ability  to  explain  the  KPA  and  its  intent,  and  to  provide  examples  relevant  to 
the  appraised  entity. 

•  knowledge  of  process  assessments.  All  team  members  must  complete  CBA  IPI  team 
training  prior  to  the  assessment.  Team  members  with  previous  experience  in  assessments 
will  provide  additional  strength  to  the  team. 

•  knowledge  of  software  process  improvement  concepts.  All  team  members  must  be 
knowledgeable  in  software  process  improvement  concepts. 

•  software  engineering  field  experience.  The  team  must  have  a  minimum  of  25  years  of 
combined  field  experience.  The  average  experience  for  individual  team  members  must  be 
at  least  6  years,  with  no  team  member  having  less  than  3  years  of  software  field 
experience. 

•  management  experience.  The  team  must  have  a  minimum  of  10  years  of  combined 
management  experience,  and  at  least  1  team  member  must  have  6  years  of  management 
experience. 

•  life-cycle  phases  and  functional  activities  experience.  Seventy-five  percent  of  the  team 
members  must  have  experience  in  at  least  one-third  of  the  organization’s  software  life- 
cycle  phases  and  their  associated  functional  activities.  At  least  two  team  members  must 
have  had  experience  in  each  life-cycle  activity. 

•  organizational  environment,  applications,  and  existing  software  process  knowledge.  At 
least  1  team  member  must  be  knowledgeable  in  both  the  organization’s  environment  and 
application  domain.  Site  team  members  are  important  to  the  team;  however,  team 
members  must  not  have  a  vested  interest  in  the  results,  such  as  an  software  engineering 
process  group  (SEPG)  team  leader  who  has  worked  on  the  organization’s  process 
improvement  initiative  for  the  past  2-3  years. 
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•  team  skills.  Each  team  member  must  have  good  written  and  oral  communication  skills,  the 
ability  to  facilitate  free  flow  of  information,  and  the  ability  to  perform  as  team  players  and 
negotiate  consensus. 

•  credibility.  Each  team  member  must  have  credibility  with  management,  respect  within  the 
organization,  and  the  ability  to  influence  people. 

•  motivation  and  commitment.  Each  team  member  must  demonstrate  the  motivation  to 
improve  the  software  process,  the  commitment  to  act  as  change  agent,  and  the  willingness 
to  do  what  it  takes  to  achieve  assessment  goals. 

Note  also  that  team  members  should  not  be  managers  of  one  of  the  selected  assessment 
projects  or  within  the  direct  supervisory  chain  of  any  of  the  anticipated  interviewees. 


3.3  Who  Are  Appropriate  Assessment  Participants? 

Selecting  assessment  participants  includes  selection  of  projects  and  project  leaders,  question¬ 
naire  respondents,  and  functional  area  representative  (FAR)  interviewees. 

The  number  and  characteristics  of  selected  projects,  respondents,  and  interviewees  must  be  a 
representative  sample  of  the  appraised  entity. 

3.3.1  Selecting  Projects  and  Project  Leaders 

Projects  are  selected  for  the  assessment  where  detailed  investigation  will  be  focused.  If  a 
project  is  selected  for  the  assessment,  then  the  project  leader  and  selected  members  of  the 
project  staff  will  participate  in  project  leader  and  FAR  interviews.  Projects  should  be  selected 
in  such  a  manner  that  they  are  representative  of  the  organizational  entity  being  appraised. 

Factors  to  be  considered  in  project  selection  include  how  closely  the  selected  projects  repre¬ 
sent  variations  in  the  assessed  entity's  projects  in  the  following  areas:  application  type  and 
domain,  technology  employed,  and  scope  in  terms  of  product  size,  staff  size,  duration,  and 
life-cycle  phases.  Normally  the  projects  selected  should  not  have  durations  less  than  six 
months.  Selected  projects  should  be  at  varying  points  in  the  life  cycle  with  emphasis  on  later 
stages.  In  addition,  project  selection  should  take  into  account  the  impact  of  the  project  on  the 
business  in  terms  of  revenue,  profit,  or  strategic  value. 

The  portion  of  the  appraised  entity  represented  by  the  projects  both  in  terms  of  number  of 
projects  and  number  of  staff  affect  the  confidence  of  the  results.  The  recommendation  is  to 
select  four  projects. 

The  goal  of  selecting  project  leader  interviewees  is  to  select  a  group  of  people  that  is  a  repre¬ 
sentative  sample  of  the  appraised  entity's  management  staff  at  the  project  level  and,  in  some 
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instances,  below.  The  assumption  is  that  this  goal  is  achieved  by  selecting  projects  that  are  rep¬ 
resentative  of  the  organization  and  interviewing  their  management  staff. 

Selected  project  leaders  must  be  committed  to  the  assessment  goals  and  the  software  process 
improvement  initiative. 

Generally  the  project  leader  interviewees  will  include  individual  leaders  of  the  projects 
selected  for  the  assessment.  However,  in  certain  cases,  the  project  leader  may  be  removed 
from  the  day-to-day  management  of  software  development  with  software  development 
responsibilities  distributed  between  several  lower-level  managers.  In  such  cases,  the  project 
leader  may  be  selected  in  addition  to  a  combination  of  functional  managers,  such  as  the  soft¬ 
ware  manager,  the  configuration  manager,  the  test  manager,  etc. 

The  leader  of  each  project  is  interviewed  individually.  If  more  than  one  individual  from  the 
project’s  management  chain  is  involved  in  the  interview,  then  none  of  the  individuals  should 
be  in  the  reporting  chain  of  any  of  the  others. 

3.3.2  Selecting  Questionnaire  Respondents 

Selection  of  maturity  questionnaire  respondents  must  be  balanced  against  the  value  of  the 
questionnaire  data  versus  the  interruption  to  the  organization.  The  questionnaire  will  be  used 
primarily  for  the  purpose  of  refining  the  data-gathering  strategy  for  the  on-site  activities. 
Questionnaire  data  will  be  corroborated  through  document  review  and  interviews:  it  is  not  the 
sole  basis  for  making  judgments  concerning  software  process  capability.  Given  these  facts, 
widespread  administration  of  the  questionnaire  may  not  be  advantageous.  The  recommended 
number  of  questionnaire  respondents  is  between  4  and  10. 

Additional  factors  to  consider  in  selecting  respondents  include  the  extent  to  which  they  repre¬ 
sent  the  selected  project  leaders  and  lead  technical  staff.  At  a  minimum  the  project  leaders  of 
the  selected  projects  should  provide  answers  to  the  questionnaire.  In  addition,  key  technical 
personnel  or  functional  managers  for  the  organization  may  be  selected  to  answer  the  question¬ 
naire.  Inclusion  of  such  additional  respondents  may  be  of  value  in  organizations  where  the 
project  leaders  are  not  involved  in  day-to-day  software  development  activities  or  where  these 
activities  are  the  responsibility  of  multiple  functional  managers. 

3.3.3  Selecting  Functional  Area  Representatives  (FARs) 

The  goal  of  selecting  FAR  interviewees  is  to  select  a  group  of  people  that  forms  a  representa¬ 
tive  sample  of  the  appraised  entity’s  technical  staff.  FAR  interviewees  should  be  practitioners, 
not  managers  or  staff.  They  should  include  opinion  leaders  and  their  managers  should  be  com¬ 
mitted  to  software  process  improvement.  It  is  desirable  to  have  FARs  who  have  the  interper¬ 
sonal  skills  required  to  facilitate  a  free  flow  of  information  during  the  FAR  interview  sessions. 
The  FARs  as  a  group  must  have  characteristics  or  attributes  (e.g.,  experience)  that  closely 
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resemble  those  of  the  appraised  entity’s  population.  The  FAR  interview  sessions  should 
include  persons  from  each  of  the  projects  that  were  selected  for  specific  investigation,  as  well 
as  persons  from  other  projects. 

No  two  individuals  who  have  a  reporting  relationship  to  each  other  should  be  in  a  FAR  inter¬ 
view  session  together.  Generally  each  FAR  session  should  have  four  to  eight  participants. 
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4  CBA  IPI  Method  Activities 


The  CBA  IPI  method  consists  of  three  phases.  The  first  phase  includes  the  activities  necessary 
to  plan  and  prepare  for  the  assessment.  The  second  phase  consists  of  on-site  activities  for  con¬ 
ducting  the  assessment,  including  techniques  for  gathering,  organizing,  and  consolidating  data. 
The  final  phase  is  to  report  the  results.  Each  phase  is  described  in  the  following  sections. 


4.1  Plan  and  Prepare  the  Assessment 

This  section  describes  the  activities  that  take  place  before  the  on-site  period.  These  activities 
are  shown  in  Figure  2. 


Figure  2:  Pre-Onsite  Activities 


4.1 .1  Identify  Assessment  Scope 

The  purpose  of  this  activity  is  to  develop  a  common  understanding  between  the  assessment 
team  leader  and  the  sponsor  concerning  the  assessment  goals,  scope,  constraints,  roles,  respon¬ 
sibilities,  and  outputs  and  to  obtain  commitments  to  proceed  with  the  assessment. 
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4.1.2  Develop  Assessment  Plan 


A  plan  for  conducting  the  assessment  is  developed  based  on  the  assessment  goals  identified  by 
the  sponsor.  Included  in  the  plan  are  detailed  schedules  for  the  assessment  as  well  as  any  risks 
identified  with  execution  of  the  assessment.  During  this  activity,  a  site  information  packet  can 
be  developed  that  will  assist  the  assessment  team  in  understanding  the  assessed  organization. 
Assessment  team  members,  projects,  and  assessment  participants  are  selected  according  to 
defined  criteria.  Documents  are  identified  for  initial  review,  and  the  logistics  for  the  on-site 
visit  are  identified  and  planned. 

4.1 .3  Prepare  and  Train  Team 

During  this  activity,  the  assessment  team  leader  must  ensure  that  each  assessment  team  mem¬ 
ber  has  received  adequate  CMM  training  (at  least  equivalent  to  the  three-day  SEI  Introduction 
to  the  CMM  course)  prior  to  being  trained  in  the  CBA IPI  method.  The  team  leader  then  con¬ 
ducts  the  three-day  CBA  IPI  Team  Training  so  that  each  team  member  understands  the  assess¬ 
ment  process,  its  underlying  principles,  the  tasks  necessary  to  execute  it,  and  each  person’s 
role  in  performing  the  tasks.  Team  training  is  followed  by  planning  sessions  that  include  estab¬ 
lishing  ground  rules  for  the  assessment,  discussion  of  the  details  of  the  assessment,  and  the 
scripting  of  interview  questions. 

4.1 .4  Brief  Assessment  Participants 

The  purpose  of  this  activity  is  to  ensure  that  the  assessment  participants  understand  the  assess¬ 
ment  process  and  have  a  clear  set  of  expectations  regarding  its  outcomes. 

4.1.5  Administer  Questionnaire 

Information  about  the  organization’s  software  processes  is  collected  from  selected  members  of 
the  organization  using  the  SEI  maturity  questionnaire.  This  information  is  used  to  provide 
team  members  an  overview  of  the  organization’s  process  capability;  often,  comments  about 
the  questions  provide  more  value  to  the  team  than  the  yes/no  responses. 

4.1 .6  Prepare  for  Onsite 

There  are  several  activities  that  must  be  done  during  the  pre-onsite  period  to  make  the  onsite 
activities  more  efficient  and  effective.  The  more  that  can  be  done  during  the  pre-onsite,  the 
more  reasonable  will  be  the  time  schedule  for  the  onsite  activities.  The  following  activities 
should  be  done  by  the  assessment  team  in  preparation  for  the  onsite  assessment. 


24 


CMU/SEI-2001-TR-033 


•  Examine  Questionnaire  Responses.  The  assessment  team  members  examine  the  pattern 
and  nature  of  responses  on  the  maturity  questionnaires  completed  by  site  personnel.  The 
responses  assist  the  team  in  focusing  their  investigations. 

•  Conduct  Initial  Document  Review.  An  initial  set  of  documents  about  the  organization’s 
software  processes  are  reviewed  to  find  additional  areas  for  probing,  to  understand  the  life 
cycle(s)  in  use  by  the  organization,  and  to  map  organization  data  to  the  CMM. 

•  Script  Interview  Questions.  The  purpose  of  this  activity  is  to  prepare  for  efficient 
interviews  conducted  during  the  on-site  period  to  maximize  collection  of  relevant  data. 

4.2  Conduct  the  Assessment 

This  section  describes  the  activities  that  take  place  during  the  on-site  period.  These  activities 

are  illustrated  in  Figure  3. 


4.2.1  Conduct  Opening  Meeting 

This  activity  kicks  off  the  on-site  visit  and  sets  participants’  expectations  concerning  the 
assessment  activities.  The  sponsor  of  the  assessment  opens  the  presentation  to  show  visible 
support  and  urge  participants  to  be  forthcoming  in  interview  sessions. 

4.2.2  Interviews 

The  purposes  of  interviewing  are  to  identify  areas  that  people  believe  can  and  should  be 
improved  in  the  organization,  to  understand  how  the  work  is  performed,  to  understand  the  pro¬ 
cesses  in  use,  to  understand  the  relationships  among  the  organization-level  processes  and  the 
project-level  processes,  and  to  ensure  coverage  of  the  CMM  within  the  scope  of  the  assess¬ 
ment  across  the  organization,  as  defined  in  the  scope. 

•  Project  leaders  are  individually  interviewed  by  the  assessment  team.  Typically,  four 
representative  projects  are  selected,  and  their  project  leaders  participate  in  these  in-depth 
interviews.  These  interviews  generally  address  issues  associated  with  project  management 
and  the  processes  in  use  on  the  project. 

•  Middle  managers  are  interviewed  as  a  group  to  understand  the  middle  management 
perspective  of  how  the  work  is  performed  in  the  organization,  any  problem  areas  of  which 
they  are  aware,  and  improvements  that  they  feel  need  to  be  made. 

•  Functional  area  representatives  are  interviewed  to  collect  data  within  the  scope  of  the 
assessment  and  to  identify  areas  that  the  software  practitioners  believe  can  and  should  be 
improved  in  the  organization.  Typical  functional  area  representative  sessions  would 
include  6-12  persons  grouped  by  areas  such  as  SEPG,  requirements,  design,  code  and  unit 
test,  integration,  and  system  test  activities. 
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Day  1  - ►  Day  N 

Figure  3:  Chronology  of  On-Site  Activities 

4.2.3  Consolidate  Information 

The  purpose  of  this  activity  is  to  summarize  and  consolidate  information  into  a  manageable  set 
of  observations  which  are  categorized  with  reference  to  the  KPAs  of  the  CMM.  All  observa¬ 
tions  must  be  validated  using  the  rules  of  corroboration.  The  team  must  reach  consensus  on  the 
validity  of  observations  and  whether  sufficient  data  in  the  areas  being  investigated  have  been 
collected.  It  is  the  team’s  responsibility  to  obtain  sufficient  information  to  cover  the  organiza¬ 
tion,  the  software  development  life  cycle,  and  the  CMM  components  within  the  assessment 
scope  before  any  rating  can  be  done.  The  responsibilities  for  guiding  the  consolidation  efforts 
for  individual  KPAs  are  usually  divided  among  team  members. 


4.2.4  Prepare  Draft  Findings  Presentation 

Draft  findings  are  generated  from  the  observations  in  preparation  for  obtaining  feedback  from 
the  assessment  participants  who  have  provided  the  information  through  interviews.  Ratings 
are  not  considered  until  after  the  draft  findings  presentations,  since  the  assessment  team  is  still 
collecting  data. 

4.2.5  Present  Draft  Findings 

The  purpose  of  this  activity  is  to  obtain  feedback  from  the  assessment  participants  about  the 
assessment  team’s  draft  findings.  Feedback  is  recorded  for  the  team  to  consider  at  the  conclu¬ 
sion  of  all  of  the  draft  findings  presentations.  Draft  findings  are  presented  in  multiple  sessions 
to  protect  the  confidentiality  of  the  assessment  participants. 
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4.2.6  Consolidate,  Rate,  and  Prepare  Final  Findings 


The  assessment  team  consolidates  additional  data  obtained  during  the  draft  findings  presenta¬ 
tions  and  any  follow-up  interviews  and  document  review.  When  the  team  has  achieved  full 
coverage  of  the  CMM,  the  organization,  and  the  software  life  cycle,  the  rating  process  may 
begin  by  rating  each  goal  for  each  KPA  within  the  assessment  scope.  Ratings  are  always  estab¬ 
lished  based  on  consensus  of  the  entire  assessment  team.  For  each  goal,  the  team  reviews  all 
weaknesses  that  relate  to  that  goal  and  asks:  “Is  there  a  weakness  significant  enough  to  have  a 
negative  impact  on  the  goal?”  If  so,  the  goal  is  rated  unsatisfied.  If  the  team  decides  that  there 
are  no  significant  weaknesses  that  have  an  impact  on  a  goal,  it  is  rated  satisfied.  For  a  KPA  to 
be  rated  satisfied,  all  goals  for  the  KPA  must  be  rated  satisfied.  A  KPA  may  be  rated  satisfied, 
unsatisfied,  not  applicable,  or  not  rated.  Assignment  of  a  maturity  level  rating  is  optional  at  the 
discretion  of  the  sponsor.  For  a  particular  maturity  level  rating  to  be  achieved,  all  key  process 
areas  within  and  below  a  given  maturity  level  must  be  satisfied.  For  example,  for  an  organiza¬ 
tion  to  be  rated  at  maturity  level  3,  all  KPAs  at  level  3  and  at  level  2  must  have  been  investi¬ 
gated  during  the  assessment,  and  all  KPAs  must  have  been  rated  satisfied  by  the  assessment 
team.  The  final  findings  presentation  is  developed  by  the  team  to  present  to  the  sponsor  and 
the  organization  the  strengths  and  weaknesses  observed  for  each  KPA  within  the  assessment 
scope,  the  ratings  of  each  KPA,  and  the  maturity  level  rating  if  desired  by  sponsor. 


4.3  Report  Results 

4.3.1  Present  Final  Findings 

The  team  leader  (or  designated  presenter)  presents  the  assessment  results  to  the  sponsor.  The 
sponsor  owns  the  assessment  results  and  is  free  to  use  them  as  he  or  she  sees  fit.  During  the 
final  findings  presentation,  the  presenter  must  ensure  that  the  organization  understands  the 
issues  that  were  discovered  during  the  assessment  and  the  key  software  process  issues  that  it 
faces.  Organizational  strengths  are  presented  to  validate  what  the  organization  is  doing  well. 
Strengths  and  weaknesses  are  presented  for  each  KPA  within  the  assessment  scope  as  well  as 
any  non-CMM  issues  that  affect  process.  A  KPA  profile  is  presented  showing  the  individual 
KPA  ratings. 

4.3.2  Conduct  Executive  Session 

The  purpose  of  this  activity  is  to  allow  the  senior  site  manager  to  clarify  any  issues  with  the 
assessment  team,  to  confirm  his  or  her  understanding  of  the  software  process  issues,  and  to  get 
guidance  regarding  the  focus,  timing,  and  priorities  of  the  recommendations  report  and  follow- 
on  activities. 
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4.3.3  Wrap-Up  of  the  Assessment 

The  team  leader  collects  feedback  from  the  assessment  participants  and  the  assessment  team 
on  the  assessment  process,  collects  information  that  needs  to  be  reported  to  the  SEI,  and 
assigns  responsibilities  for  follow-on  activities. 
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5  Time  Frame  and  Resource 
Requirements 


To  identify  the  resources  that  are  needed  to  perform  an  assessment,  a  typical  time  frame  along 
with  people  required  are  discussed  in  the  following  subsections. 


5.1  Assessment  Time  Frame 

A  typical  assessment  time  frame  is  shown  in  Table  1. 

Table  1:  Typical  Assessment  Time  Frame 


months  1  -2 

assessment  planning  and  team  training 

month  3 

on-site  assessment 

month  4 

final  report  delivery  and  recommendations  briefing 

month  5 

action  plan  development 

months  6-24 

action  plan  implementation 

months  12-30 

re-assessment 

5.2  Resource  Requirements 

Resources  required  for  an  organization  to  perform  a  CBA IPI  vary  according  to  the  scope  of 
the  assessment.  The  numbers  in  Table  2  may  be  used  as  guidelines  in  planning  the  resource 
requirements  for  an  assessment.  These  figures  were  gathered  from  early  implementations  of 
CBA  IPI  assessments.  However,  subsequent  assessments  have  shown  that  the  assessments  can 
be  done  more  efficiently,  so  these  estimates  are  considered  to  be  conservative  and  on  the  high 
end. 
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Table  2:  Resource  Requirements  for  CBA  IPI  (based  on  early  trial  data) 


Assessment  Phase 

Resource  Required 

Full-time  Equivaient 

Planning 

Assessment  team  leader 

Sponsor 

Site  coordinator 

1 0-20  days 

3-5  days 

1 0-20  days 

Pre-Onsite  Activities 

Assessment  team  leader 

Site  coordinator 

Assessment  team  members 

10-14  days 

1 0-1 4  days 

6-8  days  each 

On-Site  Activities 

Assessment  team  leader 

Sponsor 

Site  coordinator 

Assessment  team  members 

Assessment  participants  (project  leaders, 
middle  managers,  and  functional  area 
representatives) 

6-10  days 

4- 5  hours 

6-10  days 

5- 10  days  each 

1.5  days  each 

Post-Assessment 

Activities 

Assessment  team  leader 

Sponsor 

Site  coordinator 

Assessment  team  members 

4-8  days 

1  day 

2  days 

1  -2  days  each 

Totals 

Assessment  team  leader 

Sponsor 

Site  coordinator 

Assessment  team  members 

Assessment  participants  (project  leaders, 
middle  managers,  and  functional  area 
representatives) 

30-52  days 

8-11  days 

28-46  days 

1 2-20  days  each 

1 .5  days  each 

In  the  November  2001  analysis  of  user  feedback,  the  following  data  (see  Figure  4)  was 
reported  for  days  spent  in  onsite  activities  from  data  for  260  CBA  IPI  assessments. 
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Figure  4:  Number  of  Assessment  Days  Onsite 


30 


CMU/SEI-2001-TR-033 


There  is  an  inclination  to  attempt  to  complete  the  onsite  activities  during  a  five-day  period  due 
to  resource  availability.  This  is  satisfactory  when  only  a  portion  of  the  reference  model  is 
being  investigated,  e.g.,  KPAs  for  maturity  level  2.  However,  when  the  model  scope  exceeds 
the  maturity  level  2  KPAs,  it  is  usually  necessary  to  extend  the  onsite  period.  The  assessment 
must  be  planned  based  on  several  variables:  scope  of  the  organization  being  assessed,  number 
of  interviews  planned,  number  of  KPAs  to  be  investigated,  and  experience  level  of  the  team 
members. 

The  number  of  hours  that  the  team  members  worked  together  performing  the  onsite  activities 
was  reported  in  the  November  2001  analysis  of  user  feedback  and  produced  the  data  in 
Figure  5. 


Figure  5:  On-Site  Activity  Hours 


Approximately  33  percent  of  the  on-site  activities  were  conducted  in  5  days.  Forty-nine  per¬ 
cent  were  conducted  in  6  to  10  days.  The  median  numbers  of  hours  that  the  assessment  teams 
worked  together,  i.e.,  clock  hours,  for  the  assessment  activities  are: 

•  Pre-on-site  Activities:  37  hours 

•  On-site  Activities:  62  hours,  which  includes  a  median  of  20  hours  for  consolidation  of  data 

•  Total:  96  hours. 
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6  Follow-On  Activities 


To  benefit  from  the  assessment,  the  organization  must  have  an  improvement  infrastructure  in 
place.  This  infrastructure  would  include  management  sponsorship,  establishment  of  an  SEPQ 
and  CMM  training  for  the  organization. 

The  assessment  team  will  develop  recommendations  and  document  the  assessment  results 
immediately  following  the  final  findings  briefing.  Although  a  final  report  is  optional  but  rec¬ 
ommended,  if  a  final  report  is  not  required  by  the  sponsor,  one  of  the  team  members  should  be 
responsible  for  collecting  the  assessment  details,  removing  all  attribution,  and  making  them 
available  to  the  action  planning  team.  These  assessment  details  are  critical  for  prioritizing 
issues,  developing  the  action  plan,  and  implementing  the  establishing  phase. 

There  are  several  benefits  to  the  establishing  phase:  it  provides  an  organized  and  planned 
approach  to  software  process  improvement;  a  clear  understanding  of  costs,  schedules,  and 
resources;  and  the  opportunity  to  plan  for  actions  that  the  organization  has  the  capability  to 
achieve.  It  is  important  to  move  into  the  establishing  phase  as  soon  as  possible  after  comple¬ 
tion  of  the  assessment  so  that  momentum  from  the  assessment  can  be  maintained  and  the 
staff’s  expectations  can  be  managed.  An  assessment  provides  expectations  for  improvement  in 
an  organization;  follow-on  activities  are  very  important  to  avoid  disappointment  or  disillusion¬ 
ment  by  the  assessment  participants. 
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Appendix  A  SEI  Appraiser  Program 


SEi  Appraiser  Program 

The  SEI  Appraiser  Program  is  designed  to  maintain  the  quality  of  participants  in  appraisal 
technology  within  the  software  community.  The  goals  of  the  program  are 

•  to  maximize  the  value  and  use  of  SEI  appraisal  methods,  designed  and  facilitated  by 
qualified,  trained  individuals,  as  part  of  a  systematic  improvement  program  within 
organizations  that  produce  software 

•  to  transition  appraisal  technology  to  SEI  clients  in  an  effective  manner,  maintaining 
consistency  and  quality  in  the  process. 

The  SEI  strives  to  ensure  the  continued  confidence  of  the  software  community  in  the  quality  of 
SEI  process  appraisal  technologies.  The  SEI  Appraiser  Program  selects  and  trains  the  highest 
quality  candidates  to  lead  appraisals.  Persons  meeting  the  requirements  of  the  program  have 
credentials  that  distinguish  them.  They  have  access  to  SEI  appraisal  methods,  training  materi¬ 
als,  technical  support,  and  upgrade  training.  Through  their  participation  in  appraisals  and 
through  feedback  mechanisms  built  into  appraisal  methodologies,  they  participate  in  the 
advancement  of  appraisal  technologies.  The  Appraiser  Program  is  intended  in  time  to  encom¬ 
pass  multiple  appraisal  methods,  specifically,  assessments  and  evaluations.  Lead  Assessors  are 
authorized  within  the  program  to  perform  assessments.  Lead  Evaluators  are  authorized  to  con¬ 
duct  evaluations.  SCAMPI  Lead  Assessors  are  authorized  to  conduct  SCAMPI  assessments 
using  the  CMM  Integrated  models. 


Lead  Assessors 

Lead  Assessors  are  authorized  to  market  and  perform  assessments  either  for  third-party  orga¬ 
nizations  or  for  their  own  organizations’  internal  use.  Current  Lead  Assessors  are  trained  in  the 
CMM-Based  Appraisal  for  Internal  Process  Improvement  (CB  A IPI).  The  CBA IPI  method  is 
an  SEI  product  that  provides  the  software  community  a  reliable  assessment  of  an  organiza¬ 
tion’s  software  process  and  provides  guidance  for  improving  the  process. 
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Lead  Assessors  commit  to 


•  sign  a  letter  of  agreement  with  the  SEI  that  identifies  Lead  Assessor  responsibilities 

•  verify  that  assessment  team  members  have  met  the  CMM  training  requirement 

•  conduct  CBA  IPI  Team  Training  for  assessment  teams.  Training  teams  to  conduct 
independent  assessments  without  a  Lead  Assessor  is  not  permitted. 

•  participate  in  at  least  two  CBA  IPI  assessments  within  each  24-month  authorization  period 
and  must  lead  at  least  one  of  the  two  required  assessments,  and  file  a  complete  assessment 
report  with  the  SEI  for  each  completed  assessment 

•  obtain  and  use  SEI  copyrighted  materials  for  CBA  assessments 

•  use  a  new  Lead  Assessor’s  kit  for  each  assessment  or  purchase  a  Lead  Assessor  multi¬ 
assessment  kit  according  to  a  quantity  price  (with  permission  to  reproduce  these  materials 
for  a  designated  period  of  time) 

•  complete  successfully  any  required  upgrade  courses  and  examinations,  and  accept  and  use 
upgraded  materials  when  available 

•  cooperate  in  random  audits  of  assessments  by  the  SEI  and  take  any  recommended 
remedial  action  as  a  result  of  the  audit 

Lead  Assessors  are  encouraged  to 

•  present  educational  assessment  awareness  classes 

•  observe  candidate  Lead  Assessors  and  complete  observations  reports,  provided  the 
qualifications  for  an  Observing  Lead  Assessor  are  met 


Becoming  a  Lead  Assessor 

To  participate  in  the  SEI  Appraiser  Program  and  become  an  authorized  Lead  Assessor,  appli¬ 
cants  must  meet  certain  prerequisites: 

•  verified  participation  as  an  assessment  team  member  in  at  least  2  CBA  IPIs  within  the 
prior  24-month  period  (SEI  verifies  from  the  Process  Appraisal  Information  System) 

•  minimum  of  10  years  of  software  engineering  experience  in  software  development  or 
maintenance  in  an  appropriate  technical  area,  e.g.,  software  design,  software  quality 
assurance,  requirements  analysis,  configuration  management,  and  testing 

•  minimum  of  two  years  of  experience  managing  software  development  personnel 

•  advanced  degree  or  equivalent  experience  in  an  appropriate  technical  area 
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successful  completion  of  the  SEI  course  Introduction  to  the  CMM 


It  is  also  recommended  that  applicants  have 

•  oral  and  written  communication  skills 

•  the  ability  to  interact  with  management  and  members  of  the  technical  staff 

•  demonstrated  knowledge  and  appreciation  of  software  process 

•  good  technical  and  instructional  presentation  training  skills 

•  the  ability  to  work  effectively  in  a  team  environment  and  knowledge  of  group  facilitation 
and  team-building  techniques 

•  knowledge  of  total  quality  management  and  quality  improvement  techniques 

Once  the  prerequisites  are  met,  applicants  must  submit  an  SEI  Appraiser  Program  application, 
a  software  process  background  resume,  and  a  registration  form  to  attend  an  offering  of  CBA 
Lead  Assessor  Training  at  the  SEI.  At  the  successful  completion  of  training,  the  applicant  is 
considered  to  be  a  candidate  Lead  Assessor. 

A  candidate  Lead  Assessor  must  perform  as  a  CBA  IPI  team  leader,  under  the  observation  of  a 
qualified  Observing  Lead  Assessor,  within  12  months  after  successful  completion  of  training. 
A  candidate  Lead  Assessor  must  receive  a  satisfactory  observation  report  and  recommenda¬ 
tion  from  the  observing  Lead  Assessor  to  become  fully  authorized. 


Authorization  Terms  for  Lead  Assessors 

Authorization  as  a  Lead  Assessor  is  valid  for  a  two-year  period.  Authorization  must  be 
renewed  and  a  renewal  fee  paid  every  two  years.  Renewal  is  not  automatic.  Lead  Assessors 
must  satisfactorily  fulfill  their  responsibilities  during  the  previous  two  years  to  be  renewed  for 
another  two-year  period. 

The  following  factors  are  considered  during  the  renewal  process: 

•  reports  from  audits  that  may  have  taken  place  during  the  authorization  period 

•  reports  from  assessment  clients  returned  to  the  SEI  during  the  authorization  period 

•  review  of  the  actual  assessment  data  to  determine  that  the  prescribed  process  was 
followed. 
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Qualifications  for  an  Observing  Lead  Assessor 

For  an  authorized  Lead  Assessor  to  serve  as  an  Observing  Lead  Assessor  and  observe  a  Candi¬ 
date  Lead  Assessor  while  leading  a  CBA  IPI,  the  following  qualifications  must  be  met: 

•  The  Observing  Lead  Assessor  must  be  in  good  standing  in  the  SEI  Appraiser  Program 
prior  to  assuming  the  role  of  an  Observing  Lead  Assessor. 

•  The  Observing  Lead  Assessor  must  have  led  at  least  two  CBA  IPIs  since  being  fully 
authorized. 

•  The  Observing  Lead  Assessor  must  maintain  objectivity  and  have  an  independent 
relationship  to  the  Candidate  Lead  Assessor,  such  as  to  avoid  any  actual,  or  appearance  of, 
conflict  of  interest. 

•  The  Observing  Lead  Assessor  is  not  to  serve  as  a  member  of  the  assessment  team. 

Note:  The  Observing  Lead  Assessor  will  be  named  on  the  PAIS  report  and  will  receive  credit 

for  participating  on  the  assessment  as  the  team  leader. 


Contact  Information 

If  you  have  questions  or  would  like  more  information  about  the  SEI  Appraiser  Program, 
contact: 

Customer  Relations 
Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 

phone 

412/268-5800 

email 

customer-relations@sei.cmu.edu 

World  Wide  Web 
http://www.sei.cmu.edu 
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Appendix  B  Method  Roles  and 

Responsibilities 


There  are  a  number  of  roles  associated  with  the  various  assessment  tasks.  Individuals  may 
assume  multiple  roles  during  the  assessment,  and  roles  may  rotate  when  appropriate. 

Assessment  team  leader.  Experienced  individual  who  leads  the  assessment  and  keeps  the 
team  together  and  on  track.  The  assessment  team  leader  must  be  an  authorized  Lead  Assessor 
in  the  SEI  Appraiser  Program.  During  the  assessment  planning  phase  the  team  leader  is 
responsible  for  soliciting  sponsor  input,  explaining  the  impact  of  the  assessment  scope  and 
constraints  on  the  assessment  goals,  providing  cost/schedule  estimates,  setting  expectations, 
and  obtaining  a  commitment  to  proceed. 

The  team  leader  is  also  responsible  for  ensuring  that  all  pre-onsite  planning  activities  have 
been  completed  and  all  preparation  for  the  assessment  is  complete,  including  training  the 
assessment  team.  The  team  leader  may  have  assistance  from  a  qualified  person  during  presen¬ 
tation  of  the  team  training,  although  the  team  leader  has  the  responsibility  to  see  that  team 
training  is  performed  successfully.  During  the  conduct  of  the  assessment,  the  team  leader 
assigns  responsibilities  for  assessment  tasks,  facilitates  interviews,  monitors  team  member  and 
interviewee  performance,  acts  as  timekeeper  (or  assigns  one),  and  manages  adherence  to  the 
assessment  process  and  schedule. 

Assessment  team  members.  Assessment  team  members  are  required  to  have  taken  the  SEI 
Introduction  to  CMM,  or  an  equivalent  course  in  the  team  leader’s  judgment,  and  received 
CBA IPI  Team  Training  from  the  assessment  team  leader.  Each  team  member  is  responsible 
for  reviewing  the  site  information  packet  and  identifying  documents  for  initial  review.  Each 
team  member  is  also  responsible  for  asking  questions  during  interview  sessions,  reviewing 
notes,  identifying  and  classifying  significant  information  obtained  during  interviews  and  docu¬ 
ment  review,  and  identifying  additional  information  required.  Team  members  are  responsible 
for  coming  to  consensus  on  the  assessment  findings  and  ratings. 

Functional  area  representatives  (FARs).  FARs  are  site  representatives  who  are  practitioners 
and  have  technical  responsibilities  for  certain  domains.  The  FARs  are  responsible  for  attend¬ 
ing  the  assessment  participants’  briefing,  opening  meeting,  the  FAR  interview  session  to 
which  they  are  assigned,  the  draft  findings  presentation,  and  the  final  findings  presentation. 
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Middle  managers.  Middle  managers  are  site  representatives  who  fall  between  the  project 
leaders  and  the  senior  site  manager  in  the  organizational  hierarchy.  The  middle  managers  are 
responsible  for  attending  the  assessment  participants’  briefing,  the  opening  meeting,  the  mid¬ 
dle  manager  interview  session  to  which  they  are  assigned,  the  draft  findings  presentation,  and 
the  final  findings  presentation. 

Project  leaders.  Project  leaders  are  site  representatives  who  have  the  leadership  responsibility 
for  the  software  aspect  of  one  of  the  projects  selected  for  the  assessment.  The  project  leaders 
are  responsible  for  attending  the  assessment  participant’s  briefing,  opening  meeting,  the 
project  leader’s  interview  session  to  which  they  are  assigned,  the  draft  findings  presentation, 
and  the  final  findings  presentation. 

Questionnaire  respondents.  Questionnaire  respondents  are  site  representatives  who  are 
selected  to  complete  the  maturity  questionnaire.  Usually  the  project  leaders  who  have  been 
selected  for  individual  interviews  are  asked  to  complete  the  questionnaire,  and  there  may  be 
additional  questionnaire  respondents  from  the  organization. 

Senior  site  manager  or  sponsor.  The  senior  site  manager  or  sponsor  acts  as  sponsor  of  the 
assessment  and  gives  the  team  leader  commitment  to  proceed  in  the  assessment  start-up  phase. 
The  sponsor 

•  identifies  to  the  team  leader  the  business  goals  that  bear  on  the  organization’s  software 
development  and  maintenance  activity 

•  describes  to  the  team  leader  any  current  quality  improvement  initiatives  in  the  appraised 
organization 

•  identifies  the  scope  of  the  assessment  and  any  constraints  that  will  exist 

•  provides  authorization  to  proceed  and  personally  participates  in  the  opening  meeting  by 
urging  participants  to  be  forthcoming  and  supportive  of  the  assessment  effort 

•  is  responsible  for  selecting  the  assessment  team  with  selection  criteria  and  input  provided 
by  the  assessment  team  leader  (This  task  may  be  delegated  to  someone  who  is  very 
familiar  with  the  organization’s  staff  and  software  process,  and  the  assessment  process.) 

•  is  responsible  for  approving  the  assessment  plan 

•  is  the  recipient  of  the  final  findings  presentation  and  is  responsible  for  providing  resources 
for  follow-on  process  improvement  activities 

There  may  be  circumstances  where  the  sponsor  of  the  assessment  represents  a  corporate  pro¬ 
cess  improvement  organization,  and  the  senior  site  manager  is  in  charge  of  the  organization 
being  assessed.  In  that  instance,  the  team  leader  must  clarify  the  roles  between  the  sponsor  and 
the  senior  site  manager,  and  both  must  approve  the  assessment  plan. 
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Site  coordinator.  The  site  coordinator  is  responsible  for  handling  the  logistics  of  the  assess¬ 
ment.  The  site  coordinator  is  responsible  for  developing  the  schedule,  notifying  the  assessment 
participants  of  the  schedule,  making  sure  that  adequate  rooms  have  been  reserved  for  both  the 
pre-onsite  and  onsite  periods,  making  and  distributing  copies  of  the  schedules,  making  sure 
that  all  necessary  supplies  and  equipment  are  available  when  needed,  scheduling  contingency 
interviews,  requesting  additional  documentation,  and  ensuring  that  meals  are  taken  care  of. 
The  site  coordinator  needs  to  be  a  member  of  the  assessment  team. 
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Appendix  C  Glossary 


appraisal  A  diagnostic  performed  by  a  trained  team  to  evaluate  aspects 

of  an  organization’s  software  development  process,  e.g., 
CMM-Based  Appraisal  for  Internal  Process  Improvement 
(CBA IPI),  Software  Capability  Evaluation  (SCE). 

assessed  entity  The  organizational  units  to  which  assessment  outputs  apply. 

An  assessed  entity  may  be  any  portion  of  an  organization  in¬ 
cluding  an  entire  company,  a  selected  business  unit,  a  specific 
geographic  site,  units  supporting  a  particular  product  line, 
units  involved  in  a  particular  type  of  service,  an  individual 
project,  or  a  multi-company  team. 

assessment  An  appraisal  by  a  trained  team  of  software  professionals  to  de¬ 

termine  the  state  of  an  organization’s  current  software  pro¬ 
cess,  to  determine  the  high-priority  software  process-related 
issues  facing  an  organization,  and  to  obtain  the  organizational 
support  for  software  process  improvement,  e.g..  Software  Pro¬ 
cess  Assessment  (SPA),  CMM-Based  Appraisal  for  Internal 
Process  Improvement  (CBA  IPI). 

assessment  goals  The  desired  outcome  of  an  assessment  process. 

assessment  scope  The  organizational  entities  and  CMM  components  selected 
for  investigation. 

assessment  sponsor  The  individual  who  authorizes  an  assessment,  defines  its  goals 
and  constraints,  and  conunits  to  the  use  of  the  assessment  out¬ 
puts. 

CAF  compliant  appraisal  method 

An  appraisal  method  that  conforms  to  appraisal  method  re- 
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quirements  defined  in  the  CMM  Appraisal  Framework  (CAF) 
[Masters  95], 

Capability  Maturity  Model  for  Software  ( CMM) 

A  description  of  the  stages  through  which  software  organiza¬ 
tions  evolve  as  they  define,  implement,  measure,  control,  and 
improve  their  software  processes.  This  model  provides  a 
guide  for  selecting  process  improvement  strategies  by  facili¬ 
tating  the  determination  of  current  process  capabilities  and  the 
identification  of  the  issues  most  critical  to  software  quality 
and  process  improvement.  CMM  Version  1.1  is  specified  in 
CMU/SEI-93-TR-24  and  CMU/SE1-93-TR-25 
[Paulk  93a,  Paulk  93b]. 

CMM  Appraisal  Framework  ( CAF) 

A  framework  for  planning,  conducting,  and  completing 
CMM-based  appraisals. 

CMM-Based  Appraisal  for  Internal  Process  Improvement  (CB A  IPI) 

An  assessment  developed  at  the  SEI  to  determine  an  organiza¬ 
tion’s  current  state  of  the  software  development  process  to 
further  the  organization’s  own  internal  software  process  im¬ 
provement  program.  CBA  IPIs  are  based  on  the  CMM  for 
Software  V 1 . 1  and  comply  with  the  CAF. 

CMM  scope  of  the  assessment 

The  portion  of  the  CMM  used  as  a  framework  for  evaluating 
an  organization's  software  development  process  during  an  as¬ 
sessment. 

common  feature  The  subdivision  categories  of  the  CMM  key  process  areas. 

The  common  features  are  attributes  that  indicate  whether  the 
implementation  and  institutionalization  of  a  key  process  area 
is  effective,  repeatable,  and  lasting.  The  CMM  common  fea¬ 
tures  are: 

•  commitment  to  perform 

•  ability  to  perform 

•  activities  performed 

•  measurement  and  analysis 

•  verifying  implementation 
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confidentiality 


An  agreement  by  which  data  will  not  be  attributed  to  a  partic¬ 
ular  individual,  project,  or  organization. 


consensus 


consolidation 


corroboration 


coverage 


coverage  criteria 


A  method  of  decision  making  that  allows  team  members  to  de¬ 
velop  a  common  basis  of  understanding  and  develop  general 
agreement  concerning  a  decision. 

The  activity  of  collecting  and  summarizing  the  information 
provided  into  a  manageable  set  of  data,  to  determine  the  extent 
to  which  the  data  are  corroborated  and  cover  the  areas  being 
investigated,  to  determine  the  data’s  sufficiency  for  making 
judgments,  and  to  revise  the  data  gathering  plan  as  necessary 
to  achieve  this  sufficiency.  This  activity  is  repeated  daily  fol¬ 
lowing  data  collection  activities  during  the  on-site  period. 

Confirmation:  all  assessment  observations  must  be  confirmed 
by  information  from  different  sources  and  different  data-gath- 
ering  sessions  prior  to  use  as  findings. 

The  extent  to  which  data  gathered  addresses  CMM  compo¬ 
nents,  organizational  units,  and  life-cycle  phases  within  the 
scope  of  an  assessment. 

A  CMM  component  is  considered  to  be  covered  if  the  data 
gathered  relevant  to  the  component 

•  are  representative  of  the  organizational  units  within  the 
scope  of  the  assessment 

•  are  representative  of  the  life-cycle  phases  within  the  scope 
of  the  assessment 

•  address  each  of  the  key  practices  in  enough  depth  to  deter¬ 
mine  the  extent  of  their  implementation,  in  the  collective 
opinion  of  the  assessment  team 
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document  A  collection  of  data,  regardless  of  the  medium  on  which  it  is 

recorded,  that  generally  has  permanence  and  can  be  read  by 
humans  or  machines. 

findings  The  conclusions  of  an  assessment,  evaluation,  audit,  or  review 

that  identify  the  most  important  issues,  problems,  or  opportu¬ 
nities  within  the  area  of  investigation. 

functional  area  representatives  ( FARs) 

The  assessment  site  representatives  who  are  practitioners  and 
have  technical  responsibilities  for  certain  domains  within  the 
software  development  process. 

goal  A  summary  of  the  key  practices  of  a  key  process  area  that  can 

be  used  to  determine  whether  an  organization  or  project  has 
effectively  implemented  the  key  process  area.  The  goals  sig¬ 
nify  the  scope,  boundaries,  and  intent  of  each  key  process  ar¬ 
ea. 

IDEAL  approach  A  life-cycle  approach  for  process  improvement.  IDEAL 

stands  for  the  five  phases  of  the  approach:  Initiating,  Diagnos¬ 
ing,  Establishing,  Acting,  and  Leveraging. 

institutionalization  The  building  of  infrastructure  and  corporate  culture  that  sup¬ 
port  methods,  practices,  and  procedures  so  that  they  are  the 
ongoing  way  of  doing  business,  even  after  those  who  original¬ 
ly  defined  them  are  gone. 

key  practices  The  infrastructures  and  activities  that  contribute  most  to  the 

effective  implementation  and  institutionalization  of  a  key  pro¬ 
cess  area. 

key  process  area  ( KPA ) 

A  cluster  of  related  activities  that,  when  performed  collective¬ 
ly,  achieve  a  set  of  goals  considered  important  for  establishing 
process  capability.  The  key  process  areas  have  been  identified 
by  the  SEI  to  be  the  principal  building  blocks  to  help  deter¬ 
mine  the  software  process  capability  of  an  organization  and 
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understand  the  improvements  needed  to  advance  to  higher 
maturity  levels. 

maturity  level  A  well-defined  evolutionary  plateau  toward  achieving  a  ma¬ 

ture  software  process.  The  five  maturity  levels  in  the  SETs  Ca¬ 
pability  Maturity  Model  are  initial,  repeatable,  defined,  man¬ 
aged,  and  optimizing. 

maturity  questionnaire  (MQ) 

A  set  of  questions  about  the  software  process  that  sample  the 
key  practices  in  each  key  process  area  of  the  CMM.  The  ma¬ 
turity  questionnaire  is  used  as  a  springboard  to  appraise  the  ca¬ 
pability  of  an  organization  or  project  to  execute  a  software 
process  reliably. 

organization  A  unit  within  a  company  or  other  entity  (e.g.,  government 

agency  or  branch  of  service)  within  which  many  projects  are 
managed  as  a  whole.  All  projects  within  an  organization  share 
a  common  top-level  manager  and  common  policies. 

process  A  sequence  of  steps  performed  for  a  given  purpose;  for  exam¬ 

ple,  the  software  development  process  [IEEE  90].  A  set  of  ac¬ 
tivities,  methods,  and  practices  that  guide  people  (with  their 
software  tools)  in  the  production  of  software. 

process  maturity  The  extent  to  which  a  specific  process  is  explicitly  defined, 
managed,  measured,  controlled,  and  effective.  Maturity  im¬ 
plies  a  potential  for  growth  in  capability  and  indicates  both  the 
richness  of  an  organization’s  software  process  and  the  consis¬ 
tency  with  which  it  is  applied  in  projects  throughout  the  orga¬ 
nization. 


process  measurement 

The  set  of  definitions,  methods,  and  activities  used  to  take 
measurements  of  a  process  and  its  resulting  products  for  the 
purpose  of  characterizing  and  understanding  the  process. 

project  An  undertaking  requiring  concerted  effort,  which  is  focused 

on  developing  and/or  maintaining  a  specific  product.  The 
product  may  include  hardware,  software,  and  other  compo- 
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nents.  Typically  a  project  has  its  own  funding,  cost  account¬ 
ing,  and  delivery  schedule. 


rating 


requirement 


satisfied 


site 


A  characterization  of  an  organization’s  software  process  rela¬ 
tive  to  a  component  of  the  CMM. 

(1)  A  condition  or  capability  needed  by  a  user  to  solve  a  prob¬ 
lem  or  achieve  an  objective.  (2)  A  condition  or  capability  that 
must  be  met  or  possessed  by  a  system  or  system  component  to 
satisfy  a  contract,  standard,  specification,  or  other  formally 
imposed  documents.  (3)  A  documented  representation  of  a 
condition  or  capability  in  (1)  or  (2)  [IEEE  90]. 

Rating  given  to  a  CMM  component  that  is  applicable  in  an  or¬ 
ganization’s  business  environment  and  is  performed  either  as 
defined  in  the  CMM  or  with  an  adequate  alternative. 

A  geographic  location  of  one  or  more  of  an  organization’s 
units. 
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